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Having worked at all levels of the JTPA system, beginning at the local level and moving to the SDA 
and ultimately the state administrative level, I have seen at all these levels responsible managers and 
people with a genuine desire tc make the JTPA program as successful as possible. As managers we 
often have a pretty good idea whether or not our programs are working. But it is one thing to have 
a good idea that all is fine and quite another to convince those outside the program, such as the Con- 
gress, that the public is getting its money's worth from the JTPA program. 

In this era of information technology, we have little trouble thinking about the contribution 
sophisticated information systems make to the management of programs such as JTPA, But we know 
that computers do not replace good management practices or the development of policies responsive 
to the needs of the people the JTPA program was designed to help. The technology we need to think 
more about in this respect is social science research. Although managers have always had responsibility 
for judging the adequacy of their programs, thty have not always had reliable, practical tools with which 
to evaluate them. 

We now need evaluation tools not only for studying the return on the public's investment, but tools 
which help us manage our JTPA programs better. The program has passed its introductory phase. We 
are now expecting a fine-tuning, a maturing process. We need detailed analyses of each of the parts 
and processes within the JTPA system. We need to see where we are most effective and where we can 
make improvements. 

When we think of evaluation technology as a management tool, we see it as something capable of 
helping us accomplish tasks that would be very difficult or impossible to achieve without that tool. It 
should be easier for the JTPA manager to make decisions. The information that comes out of an evalua- 
tion effort should provide direction in setting policies and establishing procedures. 

But we should understand that the use of evaluation may involve change. And change brings risk. 
As we consider the risks associated with conducting evaluations, we should also remember the risks 
we take in not using what applied social research can offer us in understanding our program better. 
As Congress and the President develop plans to balance the federal budget, the pressure will be on all 
domestic programs to prove their worth. If the technology of evaluation can be used to help us, we 
should seriously consider making the best use of it we can. 

When the JTPA program came about, we were given a substantial amount of independence and con- 
trol over implementation, as well as over program monitoring and evaluation. As managers are able 
to join together in areas like evaluation, we can only strengthen our position and improve the quality 
of our service to the public, 

Isiah Turner 

Commissioner, Washington Employment Security 
National Governors Association Policy Conference 



PROJECT ABSTRACT 



The purpose of the JTPA EVALUATION DESIGN PROJECT is 
to assist states and SDAs 1n effectively addressing 
their new oversight responsibilities and opportunities 
under the Job Training Partnership Act. Based on the 
assumption that a substantial number of states and 
local service areas want to develop a more sophisti- 
cated program evaluation capability but need program- 
sped f 1c management tools for doing so, the project 
has produced a set of guides and Issue papers to 
assist them In carrying out comprehensive evaluations 
of JTPA over Its two year planning cycle. 

The guides suggest complementary approaches and 
methods for learning more about the way JTPA Is being 
carried out and Its Impact on participants and 
employers. The Issue papers recommend ways to judge 
tradeoffs between program costs and benefits , and 
Increase the utility of management Information systems 
for supporting evaluation activities. 

Cons 1 stent with the partnership principle underlying 
the Act, the project's Implementation has Involved 
funding, design and advisory partnerships whose 
unusual blend of expertise and experience has made a 
unique contribution to the products. 

Although the management tools produced In this project 
have been purposely tailored to state and local JTPA 
evaluation needs and resources , they can be usefully 
applied to other employment and training efforts and 
to other social programs. 



PREFACE 



In 1983, prior to the development of a des'ign for a state-wide JTPA 
Management Information System, the Washington Employment Security 
Department produced a concept paper on evaluation Issues In JTPA. The 
purpose was to sensitize the designers to the need for an MIS which 
served not orvly monitoring functions, but also evaluation. Based on 
this Interest In evaluation. In 1984 the Department submitted an 
unsolicited proposal to the National Conwnisslon for Employment Policy 
and the IBM Corporation for the purpose of developing evaluation tools 
which could be used by states and SDAs In evaluating their JTPA 
programs . 

In late 1984 sufficient funding was acquired from the National 
Cotmiisslon for Employment Policy (NCEP), the Corporate Support Program 
of the IBM Corporation and the Employment Security Department to 
Initiate and Implement the pr^"ect. A project team was created within 
the Department to coordinate and oversee the project, contracting was 
completed with appropriate consultants, a national advisory group was 
established and the design work began early in 1985. An additional 
fund search produced an array of valuable in-kind contributions of 
research consultation, participation by additional advisors, printing, 
computer software, and the use of facilities and equipment. 

Serving as the project's national sponsor, the NCEP viewed the project 
as a way to stimulate Interest 1n state and local evaluation consistent 
with the Act, and assist states and SDAs in developing the capability 
to produce Information which was sufficiently comparable to make a 
contribution to training policy at the national, state and local 
level. One of IBM's major funding priorities was to encourage Improved 
public policy research. The project satisfied IBM's Interest 1n 
supporting objective assessments of tax-supported social programs. The 
Washington Employment Security Department was keenly aware of the 
challenge of new gubernatorial oversight responsibilities. The need to 
Initiate a technical assistance effort to meet this challenge provided 
strong motivation to contribute to the project. 
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MENU OF EVALUATION PRODUCTS 



Consistent with the above purposes, the project has produced a number of 
specialized materials to assist states and SDAs in evaluating JTPA. 
These materials address issues pertinent to all phases of JTPA program 
evaluation: designing, planning, and implementing. Jwciy are packaged 
in an eight-volume set entitled JTPA Evaluation at the State and Local 
Level. In each set, in addition to this introductory Overview, the 
following planning and evaluation guides and evaluation issue papers are 
avai lable: 



Volume I: 
Volume II: 
Volume III; 
Volume IV: 
Volume V: 
Volume VI: 
Volume VII: 



Volume VIII: 



Overview 

A General Planning Guide (State or Local version) 

A Guide for Process Evaluations 

A Guide for Gross Impact Evaluations 

A Guide for Net Impact Evaluations 

An Implementation Manui.1 for Net Impact Evaluations 

Issues Related to Net Impact Evaluation 

A. Issues in Evaluating Costs and Benefits 

B. The Debate Over Experimental Vs. 
Quasi-Experimental Design 

HIS Issues in Evaluating JTPA 



The volumes in this set are organized to respond to the differing needs 
of both state and local users. The set includes distinct state and 
local versions of Volume II, The General Planning Guide . A fuller 
description, of each of the basic volumes is found in the final section 
of this overview. 



The above volumes are designed to offer JTPA users a fairly selective 
yet diversified menu of technical assistance products to meet a variety 
of evaluation needs and interests. Taken together, these products 
support comprehensive evaluations over the JTPA planning cycle. 
However, users may also wish to choose certain volumes from among this 
expanded menu. 



While recognizing the important differences in state vs. local 
evaluation issues, the products have also been developed with a respect 
for common evaluation needs. Including the need for both states and SDAs 
to obtain and analyze their own data and the need to compare information 
across subcontractors, SDAs and states. Although this set of evaluation 
products has been developed specifically for JTPA, the basic principles 
and methods can be applied more broadly to the evaluation of other 
employment and training programs and other social programs. 



BENEFITS OF THE PRODUCTS 

The general benefits of these evaluation products to state and local 
users are simple and direct. There are hundreds of studies of 
government-subsidized employment and training programs and thousands of 
books and articles on evaluation research methods to be applied to these 
programs. The volumes available throuqh this project are based on a 
careful review of such materials and represent a condensation of those 
that are most relevant. However, the volumes go beyond existing 
evaluation materials and studies in providing JTPA users with: 



STATE-OF-THE-ART APPROACHES TO ASSESSING JTPA PROGRAM 
OPERATIONS AND OUTCOMES 



PRACTICAL AND VALID METHODS FOR COLLECTING AND ANALYZING 
EVALUATION DATA 



TIMt-SAVING GUIDELINES FOR CARRYING OUT STEPS IN THE 
EVALUATION PROCESS 



A COMPENDIUM OF RELEVANT REFERENCES AND SOURCES OF FURTHER 
INFORMATION 



SPECIAL FEATURES OF THE PRODUCTS 

The evaluation guides and related Issue papers have been designed with 
the needs of the JTPA evaluator, program operator and policy-maker In 
mind. A number of special features have been Incorporated Into these 
evaluation products In order to make them as useful as possible to a 
wide range of users. Before describing each volume, the special 
features of the products as a whole are highlighted below. 

Orientati on To State and LocaT Users : These evaluation materials have 
been developed with a sensitivity to the differing orientations and 
priorities of both state and local JTPA users. Separate state and local 
versions of the General Planning Guide address the different planning 
contexts In which SDAs and states must operate. As a companion piece to 
A Guide for Process Evaluations , the volume's supplement outlines an 
approach to studying JTPA process Issues of particular relevance to 
state users. The remaining volumes distinguish between and respond to 
the differing state and local program environments In which evaluation 
must take place. 

Practlccil Approach : These materials attempt to balance the 
researcher's concern for scientific competence with the practitioner's 
call for practical utility In the evaluation effort. To achieve such 
balance, members of the Project's National Advisory Committee and other 
state and local JTPA practitioners have provided valuable Information 
and suggestions to project consultants. The resulting guides and issue 
papers discuss evaluation activities within a realistic program context, 
acknowledging the real world constraints and Influences affecting the 
evaluation process . 

Flexible Application : The evaluation designs presented in these 
guides offer sufficient flexibility that evaluation activities can be 
modified to meet the particular needs of a given JTPA program context. 

Comprehensiveness : Taken as a whole, the evaluation materials enable 
the user to carry out comprehensive assessments of JTPA programs. They 
allow the user to acquire several different perspectives on the same 
program within a particular time period. The user can interrelate 
information on program implementation, program outcomes, and net impacts 
to gain a wider understanding of what is happening in a program and why. 

Accessibnitv: The materials have been written and organized in a 
readily understandable fashion. To the degree possible, technical 
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language In the main text and, where necessary, 1s explicated so that 
HzueTpre!e^U^ ^^"^ *° ^""^ appreciate the concepts and 

Supplemental Planning Guide : The various evaluation guides are 
supplemented by a General Planning Guide. The guide through separate 
state and local versions assists users In planning, funding and 
developing an organizational capacity to carry out process, gross 
outcome and net Impact evaluations and to utilize their results 



PRODUCT SUMMARIES 

The summaries on the following pages are Intended to provide the user 
with more specific Information about each of these volumes. It should 
^MffJ^J fi . A Guide for Net Impact EvaTi.nnn. is 

sufficiently technical because of the statistical methods Involved that 
a practical manual. Volume VI, has been written to accompany It The 
guide and manual tend to be more appropriate for states since relatively 
large sample sizes are required for analysis. However, they are equally 
useful to larger SDAs and consortia of smaller SDAs which may want to 
Jointly study the collective net Impact of their programs. Regional 
evaluations, for example, can be very productive In providing management 
information relevant to regional labor markets. 

Also, although there Is a separate volume. Volume VII, which addresses 
the evaluation of costs and benefits at both the SDA and state level, 
this Issue Is covered In the gross and net Impact guides as well In 
this respect users benefit from three related but distinct approaches to 
this important element of program evaluation. 
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VOLUME II 



A GENERAL PLANNING GUIDE: STATE AND LOCAL VERSIONS 

PURPOSE 

To provide practical Information on ways In which state and local JTPA 
evaluation efforts can be planned, funded and maintained organizationally, 
and their results utilized for improving policies and programs. These are 
issues which cut across the specific evaluation research approaches and 
methods described in other volumes. Separate state and local planning guides 
recognize the different organizational contexts and environments of 
evaluation planning at the two levels. 

UTILITY TO USER 

Assists the user in better understanding complementary evaluation approaches 
and the different kinds of information they can provide, and suggests 
practical ways to develop and sustain fiscal, technical and organizational 
support for state and local evaluation efforts. 

ASSUMPTIONS 

States and SDAs have new oversight opportunities and responsibilities in 
JTPA, but may lack a history of evaluation experience and a current 
capability to perform evaluation tasks consistent with their new role. 
Consequently some states and local service areas need assistance In assessing 
the need for particular kinds of evaluation information, and in planning and 
supporting evaluation activities. 

FRAMEWORK 

The planning guides provide useful information on the meaning of process, 
gross impact and net impact evaluations, the content of the evaluation guides 
and issue papers, and the way the project's materials fit together to provide 
comprehensive information about JTPA. They develop an overall planning 
context and suggest a series of planning tasks which can help practitioners 
decide when and how to evaluate, what the organizational constraints and 
supports for evaluation are likely to be, what its benefits are and what it 
may cost, and how an evaluation effort can be constructed and maintained. 

The guides also address important resource planning issues such as the 
formulation of evaluation questions, the development of an evaluation plan 
the funding and staffing of evaluation activities, and the organizational 
implications of data collection. Utilizing information from interviews and 
informal discussions with numerous Federal, state and local JTPA 
administrators and operators, specific e:<amples of state and local 
orientations and experiences with respect to evaluation enhance the v-olume's 
usefulness to those who are expjcted to make decisions about evaluation, and 
those Involved in carrying out program evaluations. An extensive 
bibliography refers the user to additional sources of information on 
evaluation end evaluation planning. 

COMPLEMENTARY USE 

The planning guides uniquely complement the more technical evaluation guides 

and issue papers. In examining basic evaluation planning and Implementation 

issues within the specific context of JTPA, and in honoring the variety of 



Interests represented among potential users, the planning guides provide a 
valuable pragmatic perspective on evaluation which is directly relevant to 
JTPA practitioners considering ways to carry out their evaluation mandate. 
The guide's planning recommendations lay the necessary organizational 
groundwork for process, gross impact and net impact evaluations- 
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VOLUME III 



A GUIDE FOR PROCESS EVALUATIONS 



PURPOSE 

To obtain Information about the organizational characteristics of JTPA 
Implementation at the SDA and subcontractor level, with greatest attention 
given to those features Intended In the legislation to produce desired 
program outcomes. This Information Includes the nature of the key parts of 
JTPA organizations at these levels, and the extent tc which these parts fit 
effectively. 

UTILITY TO USER 

Assists the user in understanding how program Implementation mechanisms and 
processes work, and what thel r Impl lea t ions are for the achievement of 
program goals. It provides managers with a flexible evaluation tool for 
examining the strengths and weaknesses of local JTPA organizations in meeting 
their performance standards and achieving their legislated goals for clients. 
It helps pinpoint organizational problems and opportunities which require new 
or improved management strategies. 

ASSUMPTIONS 

While outcome assessment is Important in judging the results of JTPA 
activities, its purpose does not include an investigation of how such results 
may have been obtained. Managers need both kinds of information in order to 
identify what areas of program operation require the most attention, and to 
devise methods for achieving better results. Understanding the relationships 
among significant organizational factors which shape the implementation of 
the program enables managers to better comprehend and explain the level of 
compliance with Federal or state-adjusted performance standards, and the 
achievement of employment, earnings, and welfare goals for different client 
groups. 

The Importance of outcome evaluations is well established. Appreciating the 
value of process studies has been a more recent development. To satisfy the 
comprehensive assessment mandate in JTPA, evaluations of program 
implementation are essential. Each social program has a set of goals for 
redressing a social problem experienced by a group of individuals judged 
deserving of government assistance. Each program has an organizational 
strategy for accomplishing these goals with that group. To fine-tune such 
programs, information about both the nature of implementation and the quality 
of outcomes is needed. 
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FRAMEWORK 

The guide presents a framework for studying the organizational aspects of 
JTPA which are expected to successfully link clients with employers. Within 
thi s framework, the SDA (or the subcontractor) 1 s viewed as an 
"organizational system" composed of Interrelated parts, or "components," 
which work together to produce JTPA outcomes. The^>e components Involve both 
internal aspects of JTPA organizations, and characteristics of the 
environment in which JTPA is Implemented . Relationships among internal 
structures and processes, and between JTPA organizations and their 
environments, are perceived as having various levels of "fit," or consistency 
between parts. An effective fit is expected to Increase an organization's 
ability to achieve its goals. 



The major questions examined are whether all the necessary organizational 
components are present, how consistent they are with one another, how well 
they fit overall, and what the quality of the mesh Is between the 
organization and Its environment. The subjects studied are a range of 
participants In the organizational system and Its environment: policymakers, 
managers, service delivery staff, clients and employers. 

METHODS 

Using a model of the JTPA organizational system as a guide, the parts of the 
system and their Interrelationships are treated as the system's "structure"; 
the flows of resources and Information through the system are considered Its 
"processes." The latter are controlled by Inputs to the system In the form 
of key revenue, personnel and access decisions made at critical discretionary 
points. The outputs of the system are Its client outcomes. Inputs are 
transformed Into outputs through the organization's "conversion process," 
which consists of the organization's mission, activities, coordination effort 
and social climate. 

Each of these major components Is broken down Into Its most Important 
dimensions, and practical methods are suggested for systematically examining 
them. Ways to Integrate this Information In order to make judgments of fit 
are recommended. This Includes a discussion of ways to Identify problem 
areas and develop methods for addressing organizational problems. 

Data sources for studying program Implementation are varied, depending on the 
component or dimension studied. They range from easily accessible program 
documents and MIS data, to flexible and more structured questionnaire and 
Interview strategies Involving key actors In the organizational system and 
Its environment. Quantitative statistical methods can be used In analyzing 
the more easily quantified Information, while qualitative methods are 
recommended for drawing Inferences from other kinds of data. 

COMPLEMENTARY USE 

Process evaluations carried out prior to gross or net Impact studies can 
contribute Important Insights about the most useful Interventions, outcomes 
and client groups to study, and the most likely sources or bias In service 
delivery policies and processes which must be taken Into consideration. 
Carried out concurrently with, or following Impact evaluations, process 
studies can provide Important Information about why the observed short or 
longer term outcomes or net impacts may be occurring. This Information 
supports the replication of successful programs, and Informs managers about 
the source of problems In the less successful. 
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VOLUME III SUPPLEMENT 



SOME PROCESS ISSUES AT THE STATE LEVEL 



PURPOSE 

To obtain Information on the development and Implementation of state JTPA 
policies which help shape or otherwise affect the organizational structures 
and processes of SDAs or subcontractors. 

UTILITY TO USER 

Assists the user in understanding how the characteristics of state policies 
and the way they are carried out may Influence JTPA at the local level. 
Managers are provided with a tool for developing more effective state 
poViCles, which enhance the functioning of the statewide service delivery 
system while respecting the autonomy and diversity of SDAs. 

ASSUMPTIONS 

States receive the Federal funds supporting JTPA, and are expected to 
Institute controls over the use of these funds by SDAs, as well as to monitor 
SDA operations and program results. Therefore they are required to develop 
fiscal and nonflscal policies with respect to these privileges and 
responsibilities. Such policies then affect the way In which JTPA Is 
Implemented at the local level. Without a knowledge of the Influences 
created by these policies, process evaluations at the local level. Without a 
knowledge of the Influences created by these policies, process evaluations at 
the local level lack Important Information on the larger state environment. 
Also, states and SDAs need to be able to determine the major sources of 
problems In program Implementation at both the policy and practice level. 

FRAMEWORK 

Policy Issues In each major area of state responsibility Identified In the 
Act are examined within an organizational model of the "statewide service 
delivery system." The legislated roles and responsibilities of each 
organizational entity within this system are used to elaborate the model. 
The major features of the system are viewed as decentralization, local 
autonomy and the existence of multiple organizations: state, SDA, and 
subcontractor organizations. The characteristics of relationships among the 
latter, and the attributes of state "performance control systems" are 
Important Influences selected for study. 

Major process questions address resource allocation, control and coordination 
issues associated with the activities of State Job Training Coordinating 
Councils and programs funded by JTPA set-asides, and with state functions 
such as performance monitoring and service targeting. The statewide service 
delivery system is also analyzed in terms of the organizational constraints 
to and supports for state policy development as well as implementation 
responsive to local needs and Interests. 

METHODS 

Using a model of the policy formation and implementation process which 
sharpens its major features for analysis, guidelines are suggested for 
tracing the Influence and consequences of state policies with respect to SDA 
and subcontractor operations. Special attention is given to the constraints 
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Imposed by the characteristics of the statewide delivery system, such as the 
limitations of state performance monitoring systems, the paucity of 
administrative resources and the geographical separation of organizations 
within the system. The guidelines Involve an analysis of state policies In the 
context of a series of chronological steps In the policy process: beginning 
with problem Identification, moving to the proposal of alternative policies 
and the making of a policy decision, and ending with the Implementation of 
the policy. This process Is Illustrated using the youth dropout problem. 

COMPLEMENTARY USE 

As a supplement to Volume III, this material Is to be used In conjunction 
with the local level process guide, offering a complementary perspective on 
JTPA program Implementation. Used together the two process approaches 
provide broad Information on both SDA operations and the larger environment 
which sets the parameters for these operations. The state process supplement 
can provide Insights which help explain local level Implementation, and 
further Inform the results of outcome evaluations. Information on policy 
development can help locate the source of biases Imbedded In state level 
resource allocation and service targeting decisions, for which controls must 
be developed In Impact studies. 



17 

10 



VOLUME IV 



A GUIDE FOR GROSS IMPACT EVALUATIONS 

PURPOSE 

To obtain information on a broad range of post-program outcomes experienced 
by JTPA participants and employers, to compare outcomes for different groups 
of participants, and to analyze the impact of different service treatments or 
service providers on selected outcomes. Systematic descriptions and 
comparisons of outcomes are intended to provide timely Information which can 
form a baseline for studying stability and change in outcomes over time, and 
afford insights about causal relationships between outcomes and different 
kinds of implementation practices and service treatments, 

UTILITY FOR USER 

Assists the user in fine-tuning planning, management and service delivery at 
the state and local level by providing a knowledge base on a variety of well 
measured post-program outcomes. It develops information on those service 
strategies which appear to be most successful, and provides prompt 
information for identifying emerging problems in achieving goals which 
require more immediate managerial action. Such information assists states in 
judging relative SDA performance levels, gives SDAs a basis for making 
assessments of the comparative performance of subcontractors, and is useful 
in determining the direction of technical assistance efforts. 

ASSUMPTIONS 

Continuously updated information on a ^^ide variety of post-program outcoTies 
for clients and employers serves important management purposes. At the state 
level this information offers an empirical basis for resource allocation and 
the adjustment of SDA performance standards. Comparative information on 
outcomes can identify strong and weak areas of service delivery in local 
programs, enhance planning and encourage innovation. And it can help 
establish the legitimacy of state and local management policies and 
activities. Gross impact analysis cannot estimate the nature of cause and 
effect relationships as precisely as can experimental rese?.rch. However, the 
relative effects of different program services and implementation styles can 
be estimated with sufficient precision to offer valuable guidance to program 
nvinagers. 

FRAMEWORK 

The model developed for gross impact evaluations has a dual focus. The first 
on a description of the nature and level of participants* post-program 
labor market and related experiences, and the characteristics of outcomes for 
employers associated with JTPA. The second focus is on comparisons of the 
relative effectiveness with which different forms of program implementation 
and different service options produce desired outcomes. Alternative forms of 
agency implementation, such as policies directing service delivery, and 
alternative service strategies, such as program activities to which 
individuals are assigned, are viewed as "program variants'* which represent 
important explanatory variables in studying the meaning of gross outcomes. 
Within the latter focus, comparisons of client and employer outcomes for 
different service providers are usually more appropriate for states, whereas 
comparisons of these outcomes for different service treatments are typically 
more useful for SDAs . 
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The measurement of the key variables offers options to states and SDAs In 
terms of the types of measures and data sources that can be utilized. 
Participant service variants can Include the kinds of basic program 
activities and support services assigned to clients, as well as Important 
characteristics of these treatments. In studying outcomes for employers, 
measures can Involve the employer's perspective on participants* outcomes, in 
addition to fiscal and nonflscal costs and benefits of the program to 
employers. Measurement alternatives are discussed In terms of their 
reliability and expense. 

The subjects of study are program participants terminated In any status and 
u'easured at a follow-up point, and three kinds of employers: those providing 
services, those hiring job-ready participants who have received only job 
search assistance, and those hiring participants who have received more 
extensive services. 

METHODS 

Two sets of methods are suggested, neither of which require the use of an 
untreated comparison group. Systematic descriptive analysis yields 
percentage distributions and averages with respect to gross outcomes. 
Differential Impact analysis utilizes multivariate analysis methods to 
compare outcomes for the purpose of estimating associations between outcomes 
and Implementation practices or service Interventions. In carrying out the 
latter type of analysis, statistical controls are Introduced to reduce error 
and bias. 

Telephone surveys of participants and employers provide the data for both 
kinds of analysis. All, or a representative sample of participants are to be 
interviewed at three months. The guide explicitly addresses possible DOL 
follow-up requirements and ways to integrate additional data collection with 
them. Methods for longer term follow-up are elaborated. 

Three optional surveys are described for obtaining information on employers: 
a survey of termination employers , a survey of participating employers, and a 
multi-purpose survey combining employer and participant data . Alternatives 
are suggested for deciding the best measurement points for employer surveys. 
The Issues of sample size and differential analysis requirements at the state 
and local level are elaborated. The use of ready-made survey instruments is 
discussed, and a set of pretested surveys is provided. Both data collection 
and analysis methods are thoroughly described. Cost estimates and aids for 
implementing gross impact analyses are provided in appendices. 

CO!n>LEMENTARY USE 

Because of the wide variety of variables that can be studied, gross impact 
evaluations can help sort out the more important outcomes, implementation 
characteristics and service treatments for continuing study in future gross 
Impact analyses or net impact studies. Information produced by differential 
gross impact analyses can place process and net impact results into broader 
perspective. Process analysis can help identify the main influences within 
the JTPA organizational system affecting an extensive spectrum of client and 
employer outcomes, and suggest controls for reducing bias in gross and net 
Impact studies. Net impact evaluation can estimate the net effect of service 
Interventions on clients, while gross impact analyses can expand the 
Interpretation of such results through estimates of the differential effects 
of varying treatment and implementation strategies for employers as well as 
clients. 
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VOLUME V 



A QUIDS FOR NET IMPACT EVALUATIONS 

PURPOSE 

To obtain Information on JTPA's short and longer-term "return on Investment" 
for adults 1n Title IIA. That Is, to obtain valid estimates of the average 
net Impacts of JTPA programs on participants' post-program outcomes, 
attributing to program participation only the incremental gain in labor 
market and welfare experiences that occurs over and above what would have 
happened had these individuals not participated in the program. Valid 
estimates are defined as those which are unbiased and statistically reliable, 
and which can be generalized to the entity which is the focus of the 
evaluat1on~-1 .e. to an entire state, a large SDA, or a group of SDAs. 

UTILITY TO USER 

Assists the user in determining to what extent the employment, earnings, and 
welfare outcomes observed in the program are due to JTPA interventions or to 
other influences. 

ASSUMPTIONS 

States and SDAs will not generally be willing to Implement an experimental 
design in which eligible applicants are randomly assigned to 
treatment-control status, or to conduct follow-^up interviews with a large 
sample of participants and, comparison group members. However, the program's 
true return on Investment cannot be determined without comparing the outcomes 
of participants with the outcomes of similar individuals who have not 
received program services. Consequently, the identification and measurement 
of a reliable comparison group is critical to the evaluation of net impact. 
Given these assumptions, the approach taken in studying net Impact must be a 
practical one which utilizes existing data sources, and respects the 
constraints and resource supports in state and local environments. 

FRAMEWORK 

The framework for the analysis of net Impact Involves a study of 
relationships between JTPA interventions and outcomes which are embodied in 
the following questions: 

What is the overall net impact of JlPA programs on participants' 
post-program labor market experiences? 

How do the' net Impacts change over time? 

For which program activities (treatments) are the net Impacts the 
largest? 

Do individuals who remain in JTPA longer experience greater net gains 
in labor market outcomes? 

How does the net Impact of JTPA vary by local program environments? 

These relationships are to be analyzed separately for men and women. Within 
the two groups , the analysis is to differentiate on the basis of age, 
ethnicity, educational level and welfare recipient status. The basic program 
interventions to be studied are classroom training, on-the-job training, work 
experience and job search assistance. The participant outcomes to be studied 
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are employment status, earnings, AFDC participation status and amount of AFDC 
grant received. Controls for the environment In which JTPA Is being operated 
are local unemployment rate and urban/rural status. The possibility of 
studying the effects of different service strategies Is Included, provided 
there are significant quantifiable differences In these strategies across 
subcontractors or SDAs. 

METHODS 

A rigorous research design and methods for data collection and analysis are 
described, Including: 1) a sampling strategy; 2) procedures for 
constructing a co>Ttpar1son group and determining Its adequacy; and 3) 
techniques for estimating and adjusting for errors and biases In the 
analysis. The comparison group choseT» Is Employment Service registrants for 
whom employment and earnings Information Is available from Unemployment 
Insurance records. The method for drawing the comparison sample. Including 
the time frame for selection and the cases to be excluded. Is fully 
detailed. Recommended sources of Information on participant and comparison 
group characteristics and outcomes, rnd on service interventions, are 
existing ongoing administrative data system s: state Employment Service 
records. Unemployment Insurance wage and benefit history records, welfare 
administration grant records, the JTPA management information system, and 
data systems providing local labor market information. 

COMPLEMENTARY USE 

Information from state and local process evaluations is an Important input to 
a net impact study. In identifyinq the characteristics of participant 
selection and assignment processes, and differences in the content and 
recording of program interventions, process analysis can provide insights 
about the di recti on and magnitude of biases which could result in 
misinterpretation of net Impact findings. Process analysis can also suggest 
quantifiable variables that should be Included in net Impact studies, and 
controls useful in reducing potential biases. Process evaluations may also 
help explain net Impact results, or suggest emerging Issues that are 
Important to study using a rigorous research design. Because gross Impact 
studies utilize surveys of participants and employers, they can enrich net 
impact findings with information on a wider range of participants and 
outcomes, and a greater variety of comparisons. Data permitting, this 
information can suggest causal relationships to be studied in future net 
impact evaluations. 
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VOLUME VI 



AN IMPLEMENTATION MANUAL FOR NET IMPACT EVALUATIONS 

PURPOSE 

To provide the user with a practical step-by-step manual which Increases the 
usability of the more technical net Impact guide. 

UTILITY TO USER 

'^^^K^IS *']^ ^"l^'" J" ^^^^^^ understanding and applying the sophisticated 
methodological techniques required In making rigorous net Impact assessments. 

FRAMEWORK AND METHODS 

The manual Is organized to direct the user to the chronology of evaluation 
activities essential in obtaining valid estimates of net Impact First a 
series of overviews Is provided which alerts the user to the major questions 
to be answered, the key variables to be defined, and the kinds of data 
sources required. Second, the methodological steps Involved in sampling and 
In data collection and analysis are detailed, beginning again with an 
over. lew of the basic elements of the recommended research design. Methods 
for constructing an appropriate comparison group are explained pragmatically 
and in detail. A summary of data collection and processing tasks is followed 
by step-by-step procedures for carrying them out, including the creation and 
merging of data files for analysis purposes. Practical steps in performing 
the statistical analysis include estimation and adjustment strategies for 
dealing with selection bias. The tasks Involved in testing for differences 
between participant and comparison group members which involve multiple 
regression techniques are described. The process for obtaining estimates of 
net impact for different participant subpopulations and different 
interventions is made explicit. 

A framework and methodology for evaluating costs and benefits is an added 
feature of the implementation manual. The manual concludes with a useful 
discussion of th? complementary nature of process, gross outcome and net 
impact evaluations. 
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VOLUME VII 
ISSUES RELATED TO NET IMPACT EVALUATION 
Part A: Issues in Evaluating Costs and Benefits 

PURPOSE 

To obtain Information on the tradeoffs between program costs and program 
costs and program benefits. 

UTILITY TO USER 

Assists the user In determining the social efficiency of JTPA, which has 
Important Implications for the allocation of program resources. Such 
Information can help managers judge whether a program or activity should be 
expanded, reduced In size or scope, redirected or discontinued. 

ASSUMPTIONS 

Participants are expected to maintain. Improve or restore their capacity to 
function effectively In the labor market- -1 .g. , to Increase their human 
capital. For the cost outlay of the program to have societal value, these 
benefits must equal or exceed the total social costs. 

FRAMEWORK 

A "human capital Investment" framework Is used In comparing monetary benefits 
against monetary costs In a benefit-cost analysis, and nonmonetary benefits 
against monetary costs In an effectiveness-cost analysis. Each of these 
comparisons. In the form of ratios, are used to estimate the return on human 
capital Investment. Estimates can also be made of net present values and 
Internal rates of return on Investment using monetary benefits and costs. 

Costs and benefits are measured from three perspectives and their value 
estimated: benefits received and costs borne by program participants, 
nonpartlclpants ("taxpayers"), and society. The latter costs and benefits 
represent the broadest perspective on the overall performance or social 
efficiency of a program. 

Given time and resource constraints, It Is recommended that benefit-cost 
analyses be attempted first, since the post-program Increase In total 
before-tax earnings, as an overall summary measure of program Impact, Is 
given the greatest policy attention In JTPA. This approach can then be 
supplemented with effectiveness-cost analyses. To carry out the two kinds of 
analyses an extensive taxonomy of economic and noneconomic costs and benefits 
Is Identified for study, with an appropriate explanation of the problem of 
double-counting In analyzing relationships among benefits. 

METHODS 

Hethods are described for calculating benefit-cost and effectiveness cost 
ratios utilizing money costs. In order to estimate return on net human 
capital Investment. Analytic techniques for qualifying these calculations 
are suggested, which adjust the dollar value of costs and benefits using an 
appropriate discount rate, compare their total values, and determine the 
social efficiency of the program based on the difference In the total dollar 
value of these costs and benefits. Examples of cost estimation strategies 
are provided, and their strengths and weaknesses explored. 
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Measuring and valuing benefits and costs are complex efforts. It Is 
difficult to assign a dollar value to Important nonmonetary benefits. 
Therefore It Is recommended that one qualify costs and benefits that are 
measurable In dollars with a discussion of measurable noneconomic outcomes 
and dlfflcult-to-measure noneconomic outcomes, which affect the overall 
benefit-cost calculation. 

COMPLEMENTARY USE 

This Issue paper contributes directly to the evaluation of program Impact: to 
gross Impact studies and to the determination of net program Impact. The 
results of benefit-cost or effectiveness-cost analyses can be usefully meshed 
with the findings yielded by both Impact guides. Process evaluations can be 
helpful In Identifying sources of costii and benefits, and suggesting ways the 
less tangible ones can be measured and valued. Finally, cost-benefit studies 
can suggest process Issues which should be given more attention. 
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Part B: The Debate Over Experimental vs. 
Qua si -experimental Approaches 

PURPOSE 

To explore the strengths and weaknesses; of using experimental vs. 
quasi-experimental research design In evaluating the Impact of JT^A programs. 

UTILITY TO USER 

Assists the user In understanding the range of research designs which can be 
used In evaluating JTPA, alerting the user to the nature and sources of 
errors and biases which must be given attention In using designs along a 
contlnum from exploratory to experimental. It also helps the user to 
consider the major scientific, political and organizational tradeoffs 
Involved In selecting one design approach over another. 

ASSUMPTIONS 

The purpose of evaluation Is to come as close as possible to revealing "the 
truth" about the Implementation and Impact of social programs, particularly 
with respect to their Intended goals. Interventions and outcomes. It Is well 
accepted that the application of scientific principles and methods Is the 
evaluator's best route to this truth. However^ even the most competent and 
rigorous use of scientific method can only approximate the truth by stating 
educated probabilities that things truly are as they appear to be from the 
analysis of unbiased Information. These probabilities give us the most 
accurate estimates of the truth available to us. 

The less~than>perfect but valuable determination of reality Involves somewhat 
different tradeoffs under laboratory conditions as compared with the 
pragmatic settings of most ongoing social programs. Therefore a controversy 
persists about the optimum evaluation approach for studying programs In 
real- life policy settings at the state and local level. 

FRAMEWORK 

In this Issue paper the above controversy Is Investigated In terms of three 
sets of tradeoffs: scientific, political and organizational. As a context 
for a discussion of these tradeoffs and their Implications for evaluating 
JTPA, the general steps In the research process and their desired qualities 
are explored. Emphasis Is on the development of a research design, as It 
applies to Impact evaluations. The research design Is a plan which guides 
the sampling of those to be studied, the direction of data, the analysis of 
that data, and the drawing of useful conclusions from that analysis. 

Exploratory, descriptive, quasi-experimental and experimental research 
designs are discussed In relation to the "Ideal experiment". A summary of 
some of the major kinds and sources of bias and error arc provided to 
Illustrate the scientific tradeoffs Involved In these different approaches. 
It Is noted that while the design of the Ideal experiment overcomes most (but 
not all) of the major barriers to obtaining "the truth" about the research 
questions to be answered In an evaluation, some of Its requirements may be 
costly politically, or may not be organizationally feasible. 
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Given the political and bureaucratic barriers encountered In carrying out 
experiments. It Is suggested that a rigorous quasi-experimental approach 
which adjusts statistically for the absence of certain desirable experimental 
conditions may be more useful In evaluating the Impact of JlPA at the state 
and local level. 

COMPLEMENTARY USE 

This issue paper complements the guides for implementing Impact evaluations, 
particularly net impact studies of JTPA. In providing the context for the 
research design decisions recorflmended in these guides, this overview of the 
ongoing debate about experimentdl vs . non- experimental research strategies 
affords states and SDAs with a rationale for considering political and 
organizational factors affecting evaliiation, as well as scientific supports 
and constraints. It also snco^uage*^ the development of a new relationship^ 
between practit 'joners and researchers with respect to a greater sensitivity 
to and appreciation of the different influences which inevitably shape 
evaluation decisions. 
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HOW TO OBTAIN THE MATERIALS 



The set JTPA Evaluation at the State and Local Level , or individual volumes 
within the set can be obtained from the Washington State Employment Security 
Department. Purchasers are encouraged to order a full set at a reduced rate, 
rather than individual volumes, because of the integrated and complementary 
nature of the volumes within the set. If you have not received an order form 
with this Overview Volume and wish to place an order, please contact the 
Project at the address below: 



Deborah Feldman 
JTPA EVALUATION DESIGN PROJECT 
Employment Security Department/Regional Office 
P. 0. Box C 70732 
Seattle, Washington 98107 



(206) 545-6515 
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